1. Technical Field
The present invention relates to an image processing system and specifically to an address generator used to calculate addresses so that multiple image frames or windows are processed in parallel.
2. Description of Related Art
Reference is made to FIG. 1 (conventional art) and FIG. 2 (conventional art) which illustrate a system 16 including a camera or image sensor 12 mounted in a vehicle 18 imaging a field of view in the forward or rear direction. Image sensor 12 typically delivers images in real time and the images are captured in a time series of image frames 15. An image processor 14 is used to process image frames 15 for processing by one or more driver assistance systems.
Driver assistance systems include forward collision warning systems, lane departure warning systems, traffic sign recognition (TSR) and headlight control systems.
Lane departure warning (LDW) systems are designed to give a warning in the case of unintentional lane departure. The warning is given when the vehicle crosses or is about to cross the lane marker. Driver intention is determined based on use of turn signals, change in steering wheel angle, vehicle speed and brake activation. There are various LDW systems available. One algorithm for lane departure warning (LDW) used by the assignee/applicant (Mobileye Technologies Ltd., Nicosia, Cyprus, hereinafter “Mobileye”) of the present application is predictive in that it computes time to lane crossing (TLC) based on change in wheel-to-lane distance and warns when the time-to-lane crossing (TLC) is below a certain threshold. Other algorithms give a warning if the wheel is inside a certain zone around the lane marker. In either case, essential to the lane departure warning system is the lane marker detection algorithm. Typically, the lane markers are detected in the camera image and then, given the known camera geometry and camera location relative to the vehicle, the position of the vehicle relative to the lane is computed. The lane markers detected in the camera image are then collected over time, for instance using a Kalman filter. Wheel-to-lane marker distance may be given with an accuracy of better than 5 centimeters. With a forward looking camera, wheel-to-lane marker distance is not observed directly but is extrapolated from the forward view of the camera. The closer road markings are observed, less extrapolation is required for determining wheel-to-lane marker distance and more accurate estimates of wheel-to-lane marker distance are achieved especially on curves of the road. Due to the car hood and the location of the camera, the road is seldom visible closer than six meters in front of the wheels of the car. In some cars with longer hoods, the road visible in front of the wheels of the car is even less. Typically the lane departure warning system of Mobileye works on sharp curves (with radius down to 125 m). With a horizontal field of view (FOV) of 39 degrees of the camera, the inner lane markers are still visible on curves with a radius down to 125 meters. In order to correctly perform lane assignment on curves, lane markings are detected at 50 meters and beyond. With a horizontal field of view (FOV) of 39 degrees for the camera, a lane mark of width 0.1 meters at 50 m distance corresponds in the image plane to just under two pixels wide and can be detected accurately. The expectation from the lane departure warning systems is greater than 99% availability when lane markings are visible. Expectation with 99% availability is particularly challenging to achieve in low light conditions when the lane markings are not freshly painted (have low contrast with the road) and the only light source is the car halogen headlights. In low light conditions, the lane markings are only visible using the higher sensitivity of the clear pixels (i.e. using a monochrome sensor or a red/clear sensor). With the more powerful xenon high intensity discharge (HID) headlights it is possible to use a standard red green blue (RGB) sensor in most low light conditions.
The core technology behind forward collision warning (FCW) systems and headway distance monitoring is vehicle detection. Assume that reliable detection of vehicles in a single image a typical forward collision warning (FCW) system requires that a vehicle image be 13 pixels wide, then for a car of width 1.6 m, a typical camera gives initial detection at 115 m and multi-frame approval at 100 m. A narrower horizontal field of view (FOV) for the camera gives a greater detection range however; the narrower horizontal field of view (FOV) will reduce the ability to detect passing and cutting-in vehicles. A horizontal field of view (FOV) of around 40 degrees was found by Mobileye to be almost optimal (in road tests conducted with a camera) given the image sensor resolution and dimensions. A key component of a typical forward collision warning (FCW) algorithm is the estimation of distance from a single camera and the estimation of scale change from the time-to-contact/collision (TTC) as disclosed for example in U.S. Pat. No. 7,113,867.
Traffic sign recognition (TSR) modules are designed typically to detect speed limit signs and end-of-speed limit signs on highways, country roads and urban settings. In addition, iconic supplemental signs (e.g. rain, on exit (arrow), trailer) are typically detected and identified. Partially occluded, slightly twisted and rotated traffic signs are preferably detected. Systems implementing traffic sign recognition (TSR) may or should ignore the following signs: signs on truck/buses, exit road numbers, minimum speed signs, and embedded signs. A traffic sign recognition (TSR) module which focuses on speed limit signs does not have a specific detection range requirement because speed limit signs only need to be detected before they leave the image. An example of a difficult traffic sign to detect a 0.8 meter diameter traffic sign on the side of the road when the vehicle is driving in the center lane of a three lane highway. Further details of a TSR system is disclosed by the present assignee in patent application Ser. No. 11/951,405.
A typical automatic headlight or high/low beam control (AHC) system detects the following conditions and switches to from high beams to low beams: headlights of oncoming vehicles, taillights of preceding vehicles, street lights or ambient light indicating that high beams are not required and a low vehicle speed. The host vehicle lights are switched back to high beams when none of these conditions exist (often after a specified grace period). One approach for detecting taillights is to compare images from two sensors: one with a red filter and the second with a cyan filter. The cyan filter responds to non-red light sources and will give zero response to red light. By comparing corresponding pixels from two imaging sensors one can detect the color of the light source. The number of pixels of each color above a certain intensity is counted and if the count is above a threshold the systems switches to low beams. The use of color filters with imaging sensors may preclude the simultaneous use of the same image frames for other driver assistance applications.
A second approach for automatic high-beam control (AHC) uses an RGB sensor to give better color differentiation. Typical light sources can be located in the full CIE color space as defined by the International Commission on Illumination. This approach distinguishes between green, yellow and red lights. A powerful green traffic light is not confused with an oncoming vehicle. Since a single sensor with a color mosaic filter i.e. Bayer pattern mosaic is used, the lens is defocused so as to spread a light source over multiple pixels. The use of the color mosaic filter reduces both the effective image sensor resolution (by 50%) and the intensity response (to less than one third). The color mosaic filter may preclude the use of the same sensor for traffic sign recognition (TSR) or lane departure warning (LDW) because of the intensity response penalty.
Given that forward collision warning (FCW), traffic sign recognition (TSR) and lane departure warning (LDW) already require a high resolution monochrome sensor, a new automatic high-beam control (AHC) algorithm was developed for use with high resolution monochrome sensors as disclosed in U.S. Pat. No. 7,566,851. A number of different pattern recognition techniques are used with higher resolution monochrome imaging sensors to identify light sources instead of relying on color information. The automatic high-beam control (AHC) algorithm includes the following features: Detect bright spots in the sub-sampled long exposure image and then perform clustering and classification in the full resolution image, classify spots based on brightness, edge shape, internal texture, get further brightness information from the short exposure frames and classify obvious oncoming headlights based on size and brightness, track spots over time and compute change in size and brightness, pair up matching spots based on similarity of shape, brightness and motion, classify pairs as oncoming or taillights based on distance, brightness and color, and estimate distance and where unmatched spots might be motorcycles taillights.
U.S. patent application Ser. No. 12/573,836 describes “bundling” or parallel processing of multiple driver assistance systems (e.g. automatic high-beam control (AHC) and traffic sign recognition (TSR), lane departure warning (LDW), forward collision warning (FCW)) on a single hardware platform 16, e.g. camera 12 and processor 14. Bundling provides cost reduction and may allow more driver assistance functions to be added to the vehicle without increasing the space required beyond the windshield of the vehicle.
Different driver assistance applications have different requirements from camera 12, and associated optics. For example, a conventional automatic high-beam control (AHC) algorithm makes significant use of color information and thus requires a color sensor, while lane detection and traffic sign recognition require the extra sensitivity of a monochrome sensor for operation under low light conditions. No single gain/exposure setting is optimal for all applications and in fact, some applications (such as AHC and TSR) may each use more than one gain/exposure setting. Most driver assistance applications require processing over multiple windows of a single frame and/or multiple image frames. In some cases, image frames are partitioned between different driver assistance applications and in other cases the image frames are shared between the different driver assistance applications.
A vector processor, or array processor, is a processor that implements an instruction set containing instructions that operate on one-dimensional arrays of data called vectors. is in A scalar processor in comparison operate on a single datum. Most commonly used processors are scalar processors
VLIW (Very Long Instruction Word) digital signal processing units. contain several sub-processing units which operate in parallel. The VLIW contains multiple opcodes which direct the various sub-processing units, thus allowing for parallel processing since each sub-unit gets its opcode at the same time. An op-code is a portion of a machine language instruction that specifies an operation to be performed.
Reference is now made to FIG. 3, a system block diagram for a conventional image processing system 30. Image processing system 30 has a processor 304 connected to a register 306 via a register bus 322. A memory 302 is connected to image sensor 12 (not shown in FIG. 3) for inputting one image frame 15 at a time via an image frame bus 310. Processor 304 has multiple arithmetic and logic units (ALU) 320 and a load store unit LSU 318 connected to a very long instruction word (VLIW) 316 via data bus 314. ALU 320 and a load store unit LSU 318 are connected to register 306 by register bus 322. LSU 318 connects to and accesses memory 302 via memory bus 312.
During operation of image processing system 30, multiple arithmetic and logic units ALU 320 operate in parallel via op-codes (operation codes) provided by VLIW 316. In the case of VLIW 316, many op-codes provide multiple commands which allow parallel processing by sub-processing units ALU 320. VLIW 316 contains address information to be used by one of ALU 320 to compute the upcoming address to be then used by LSU 318 in a subsequent cycle.
Load-storage unit (LSU 318) is responsible for accessing memory 302. The address for LSU 318 generally comes from register file 306 where the address is the result of a computation performed by the other sub-units. As a result, LSU 318 must wait until the necessary address is calculated. Furthermore, ALU 320 requires at least one cycle to performing the address calculation instead of performing an algorithm specific computation.
Thus there is a need for and it would be advantageous to have a hardware platform suitable to support multiple image frames and/or multiple windows for processing in parallel multiple driver assistance applications, and specifically an address generator which calculates addresses over multiple image frames and/or multiple windows to support processing of multiple driver assistance applications using a single hardware platform.